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ABSTRACT 

Proper understanding of scaling and large-scale hydrologic processes is often not explicitly 
incorporated in the teaching curriculum. This makes it difficult for students to connect the effect 
of small scale processes and properties (like soil texture and structure, aggregation, shrinkage, 
and cracking) on large scale hydrologic responses (like watershed runoff). An instructional module 
that introduces the concept of process-based scaling, as the framework to connect hydrologic 
processes and properties at different scales, was developed and evaluated. This paper examines 
how incorporating the concept of scaling into student curriculum impacts students’ learning and 
decision making capabilities. It presents the evaluation of this module in an undergraduate envi¬ 
ronmental and natural resources engineering course. Evaluation results supported the hypothesis 
that introducing the concept of scaling and its application (using computer models) into under¬ 
graduate engineering courses enhanced students’ learning and decision making skills. Students’ 
levels of confidence in their knowledge of hydrologic systems also increased after the introduction 
of the scaling concept and following computer modeling exercises. 

Keywords: Enhanced student learning, decision making, hydrology education, scaling. 
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1. INTRODUCTION 


Hydrologic systems are mixtures of interacting processes that cut across various spatial and tem¬ 
poral scales. One of the most complex research challenges identified in the field of hydrology is the 
problem of scale (Gupta et al., 2000, Beven, 2001, Miller and Gray, 2002, Soulsby et al., 2002, Miller 
and Gray, 2008, Tartakovsky and Winter, 2008). From molecular to continental in spatial scales and 
from picoseconds to centuries in temporal scales, fundamental knowledge gaps in understanding 
hydrologic systems present major challenges for researchers and educators. 

In academia, courses usually teach hydrologic principles at different scales. For example, basic 
science courses introduce concepts (such as atomic interaction, colloid theory, diffuse double layer, 
van der Waals attraction, and Navier-Stokes’ flow equation) at the particle scale hoping that they 
will form a foundation for students’ understanding of environmental systems. Soil physics courses, 
on the other hand, introduce concepts developed at the laboratory scale like soil bulk density, soil 
texture, structure, and organic content knowing that this knowledge is necessary for understanding 
soil-water interaction. Other courses introduce concepts and processes at the field scale including 
shrinkage and swelling, cracking, and heterogeneity. At the watershed scale, courses introduce con¬ 
cepts such as runoff prediction with the assumption that students are aware of the various related 
soil properties and interactions. Then, policy courses address issues like water allocation and best 
management practices at the watershed or basin scales assuming students’ prior knowledge of 
smaller scale relevant processes as prerequisite. 

What is lacking is a method or framework that establishes the connection between different 
processes at different scales taken from different courses. Although many instructors touch 
on scaling implicitly in their courses (by explaining the connection between the concepts they 
introduce and other concepts from close scales), students remain in need of a general scaling 
framework that helps them connect the different pieces of the puzzle (in this case, the multi¬ 
scale hydrologic principles puzzle). This framework is referred to as process-based scaling in 
this paper. For example, it is because of how clay particle interact at the particle scale that they 
have different shrinkage and swelling properties at the laboratory scale, which translates into the 
development of cracks and preferential flow at the field scale, thus affecting runoff prediction 
at the watershed scale. 

Due to the absence of scaling from undergraduate and graduate education, Raia (2005) 
stressed the need for students “to undergo a fundamental shift of paradigm: from a linear 
causal thinking approach to a systems thinking approach” to account for the complexity and 
multiscale nature of environmental systems. Manduca et al. (2008) found a need to shift mod¬ 
ern geoscience education to a more quantitative rather than qualitative field. Dickerson et al. 
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(2005) assessed student knowledge of scale in groundwater systems and indicated that many 
students hold an “inappropriate conception of hydrogeologic principles” due to incomplete 
understanding of issues of scale, thus calling for particular attention to scaling in teaching 
hydrogeologic principles. 

Different methods of instruction have been used in explaining fundamental hydrologic concepts 
ranging from field observations and laboratory testing (Trop et al., 2000, Nichols et a I., 2003) 
to computer simulation models (Mohtar and Engel, 2000, Gunn et al., 2002, Li and Liu, 2004, 
Macfarlane et al., 2006). We believe that computer models offer unique tools for understanding and 
integrating the multi-scale hydrologic processes where real observations and field monitoring fail 
due to the prohibitive time and cost required. Hydrologic systems are highly complex presenting 
special challenges especially for educators helping students learn about them and/or their compo¬ 
nents (Ex.: conceptualization and computation). Students must develop a good understanding of 
these hydrologic systems if they are to apply their discipline-based knowledge effectively to make 
educated and responsible policy-decisions based on fundamental understanding of the interactions 
between the multi-scale processes involved in their hydrologic problem. Given the complexity of the 
scaling problem (as described by NRC, 1999; Gupta et al., 2000; Miller and Gray, 2002; Miller and 
Gray, 2008 and others), and the urgency and necessity for the problem to be included in student 
curricula, new delivery methods and techniques to educate students about the various scales of 
natural resources and water quality systems are needed. Additionally, a system that allows students 
to simulate the hydrologic response or impact of their decisions at the local and large scale should 
improve their ability to make such decisions. 

This paper presents a flexible educational module on multi-scale hydrology and provides results 
from an evaluation methodology to show improvements in students’ knowledge and decision 
making skills. The hypothesis tested in this research was that student self assessment, knowledge, 
decision making skills and level of confidence in dealing with hydrologic systems can be improved 
by understanding the concept of scaling through appropriate lecture material and access to suit¬ 
able hydrologic models. The module provides learners and educators in the interdisciplinary field 
of environmental and natural resources science and engineering with instructional material, case 
studies, and problems that enhance student learning of complex hydrologic systems and their 
decision making skills. These materials are interactive and Web-accessible ( http://cobweb.ecn. 
purdue.edu/-mohtar/ASABE08workshop.htm ). The Web-based modeling module was designed 
to enhance student learning and decision making by allowing students to focus their learning 
efforts on understanding the natural resource systems being simulated rather than on how to 
use the models. By doing so, students learn about the processes and the interactions of these 
processes. 
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2. METHODOLOGY 


2.1 Goals and Objectives 

The educational module was developed to address the following objectives: (1) improving the 
quality of education and instructional capability in the multidisciplinary field of hydrology; (2) as¬ 
sisting faculty preparation and enhancement for teaching scaling in hydrology; and (3) improving 
student learning and decision-making skills within the realm of complex hydrologic systems. 

Upon successful completion of the module, students should gain fundamental understanding of 
the following: 

1. How the temporal and spatial scaling of hydrologic processes affects environmental systems. 
This includes issues of scale within the soil column and at field scales through the introduction of 
the concept of pedostructure in hydrology (Braudeau and Mohtar, 2006; Braudeau and Mohtar 
2004; Braudeau et aI., 2004a; Braudeau et al., 2004b). They also include issues of scale at the 
field to watershed scales (Ex.: Bloschl and Sivapalan, 1995, Wallender and Grismer, 2002). 

2. How the hydrologic cycle is affected by variability in precipitation, evapotranspiration (ET), 
landuse, and soil type. 

3. How to simulate hydrologic responses using select hydrologic models representing different 
scales, including: (1) Kamel®, a computer hydrology model that incorporates the pedostructure 
scaling concept for field sites; and (2) a web based version of the WEPP® soil erosion model 
( http://milford.nserl.purdue.edu/wepp/abe325.php and http://topsoil.nserl.purdue.edu/nserl- 
web/weppmain/wepp.html ) that simulates hydrology and erosion on hillslopes. 

Given the interdisciplinary nature of the topic and the wide range of departments and disciplines 
involved in teaching hydrologic processes, this module was constructed to be flexible enough to 
accommodate different time allocations (varying from a single 50-minute lecture to one- and two- 
week lecture series coupled with hands-on computer labs), academic levels (undergraduate and 
graduate), and backgrounds (forestry, agricultural engineering, hydrology, agronomy, and others). 

2.2 Evaluation Procedure 

The complexities of large-scale hydrologic systems do not allow students or instructors to 
conduct quantitative, watershed-scale learning exercise within the classroom without the use of 
modeling tools. Many of these tools require complicated and time-consuming procedures to sort 
and organize input data, as well as interpret and visualize simulation results. Unfortunately, these 
constraints make the use of most research grade modeling tools by students difficult, if not impos¬ 
sible, without significant additional instruction and oversight. Therefore, it is extremely difficult to 
have a “control group” of students without the development of interfaces that hide the complexity 
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of the problem and provide them with step-by-step procedures and annotations to help learn the 
program and understand associated issues. This investigation attempted to minimize the time and 
energy that students need to invest in working with the computer model while solving a case study, 
and optimize students’ time allocation to achieve the learning goals. 

In light of the above constraints, the outcome-based evaluation procedure developed by Mohtar 
et al. (2007) was adopted to measure the effectiveness of introducing of scaling and using hydro- 
logic models on enhancing students’ learning and decision making capabilities. This procedure is 
based on the understanding of three basic organizational levels, namely: developers, instructors or 
evaluators, and learners (Mohtar et al., 2007). Figure 1 outlines the necessary steps for an effective 
interactive learning process. 

The educational module was evaluated twice over a period of two years (Figures 2 and 3). In 
both evaluations, students were asked to complete a pretest at the beginning of the first lecture 


Steps of an Effective Interactive Learning Process 


Organization Level 


Developers 


Instructors/ 

Evaluators 


Students/ 

Learners 


«--*< 3 


* 6 


• Set the learning goals in using modeling systems. 

• Set the scope of the tools developed. 


• Set the learning objectives for the specific case 
studies in which the tool is being used. 


• Develop scenario (s) for the specific learning 
objective listed in step 2. 


• Develop and conduct an instructor’s survey to set 
the benchmark on which the students' learning 
outcome will be evaluated against. 


• Conduct pre- and post- test (s) for students' 
evaluation and differential learning after (1) getting 
introduced to the scaling and modeling concepts, 
and (2) using the model tools. 

• Test should cover the learning prism to measure the 
improvement in the three modes of learning: (1) 
qualitative. (2) quantitative, and (3) idea eliciting. 


• Evaluation and feedback loop to (1) developers to 
improve the model, and (2) instructors to improve 
the scenario and supporting material. 


Figure 1: Steps for an effective interactive learning process (Adopted from Mohtar et al., 
2007 ). 
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First Lecture Hour 
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Lecture 


Presentation 
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/' Second Lecture Hour \ 



<=> 


Homework and Posttest 2 



Figure 2: Schematic of the learning module evaluation procedure used for the first 
evaluation, in Fait 2006-2007. 


to obtain a baseline of their knowledge of hydrologic/water quality systems within the emphasis 
areas of the modeling tools that are used in the module. This assessment was followed by lecture- 
type presentations that explained and introduced the concept of scaling in hydrology, hydrologic 
processes and their interactions, the pedostructure approach, and the effect of spatial and tempo¬ 
ral variability on hydrologic response. Students were then asked to complete a series of hands-on 
computer laboratory experiences using the Kamel® and/or WEPP® models to solve case studies at 
the field scale. Students then completed a posttest (same as pretest) to evaluate improvement in 
student learning and decision making skills and were then assigned a homework based on a water¬ 
shed scale case study. The posttest was administered a second time (posttest 2) after the comple¬ 
tion of the homework assignment (due one week after the end of the module) in order to measure 
any additional improvements. 


2.3 Evaluation Constructs and Instruments 

Four evaluation constructs were used in this study: enhanced learning, decision making, self as¬ 
sessment of students’ ability to better understand hydrologic systems, and level of confidence. Those 
constructs were assessed using three instruments: (1) self assessment questions; (2) enhanced learning 
and decision making questions; and (3) students’ level of confidence in answering those questions. 
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First Lecture Hour 



Second Lecture Hour and 
Lab 



Third Lecture Hour and Lab 





Figure 3: Schematic of the revised learning module evaluation procedure used for the 
second evaluation, in Fall 2007-2008. 


Enhanced learning was observed in terms of the increase in the understanding of hydrologic 
systems following the introduction of the scaling concept. It was assessed with eleven general knowl¬ 
edge questions to measure the improvement in students’ understanding of scaling in specific as well 
as hydrologic systems and responses in general (Appendix 1). Those questions were mainly about 
hydrologic systems whose answers required (or became clearer with) knowledge of scaling. 

Decision making was observed as the ability to apply the knowledge gained to evaluate simple alterna¬ 
tives and select the best among them to solve a simple group of problems. Decision making, especially 
with the complexity of large-scale hydrologic systems, requires a special set of skills and knowledge. 
Enhancing such skills requires students to be explicitly aware of: (1) necessary content and knowledge 
and (2) the decision making process. In fact, becoming aware of the content forms the foundation to 
be able to apply, synthesize, evaluate and judge the problem at hand and to come out with the best 
decision from the realm of possible solutions. In our study, we focused on (1) providing the foundational 
knowledge required for making good decisions in this complex environment and (2) discussing the 
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decision making process that must account for ali the multi-scale components and processes within the 
hydrologic system. In the pretest and posttests, questions were not necessarily designed to evaluate 
students on advanced level decision making skills as this would be difficult to change in the time frame 
in which this module was delivered. However, six questions were designed to assess if students could 
make use of the information learned to make somewhat simple decisions about some basic environmental 
problems. The improvement in students’ performance support our assertion that the module enhanced 
their foundational knowledge and increased their sense of the decision making process. 

Self assessment was observed as students’ self assessment of their ability to better understand 
hydrologic systems at various scales as well as the importance of scaling as a framework to im¬ 
prove their knowledge and abilities. It was assessed by seven self assessment statements such as 
7 understand the hydrologic process at the small field” and 7 understand the hydrologic process 
at the watershed level" to measure students’ appreciation and interest in hydrologic systems and 
issues of scale. In each assessment, students were asked to rate their opinions to each of the seven 
statements on a scale of 1 to 5 (from “strongly disagree” to “strongly agree”) thus allowing for the 
measurement of change in self assessment following the module instruction. The internal consistency 
reliability (i.e., Cronbach’s alpha) of the scale was 0.95. 

Level of confidence (LOC) was observed as students’ self assessment of how confident they were 
answering each of the 17 questions related to enhanced learning and decision making. A ranking for level 
of confidence was added next to each of the test questions to measure (on a scale of 1 to 5) students’ 
comfort level (confidence) in answering the questions in the test. Coupling the results of this new con¬ 
struct/instrument with the test scores (from enhanced learning and decision making) provided valuable 
information on the effectiveness of the educational module. For example, if LOC increased while the 
performance in answering the questions decreased, then this was an indication that the module may 
have problems with oversimplifying the concept. Also, if LOC decreased while the performance in an¬ 
swering the questions increased, then this is an indication that the module may be confusing students 
on some issues. The target is an increase in both: the performance and LOC, which would support that 
the concept of scaling has improved students’ understanding and made their confidence higher. 

In general, Bloom’s (1956) taxonomy of the cognitive domain guided test construction to reflect a 
focus on higher-order thinking skills (e.g., analysis, synthesis, and evaluation), especially as it related 
to the decision making questions. Content validity was assessed by course instructors in accordance 
with Standards of Educational and Psychological Testing (AERA, APA, & NCME, 1999). A table of 
specifications (i.e., a test blueprint) was developed by the content experts on the research team. 
This allowed for the concepts and number of items required to assess these concepts to be clear 
and appropriate given the time constraints of testing and the level of student knowledge. That is, 
the process allowed the team to align the content measuring each construct to the items (type and 
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level of difficulty) to be used. This is a typical first step in instrument development to assist with 
score validity. After two rounds of testing the instrument with students (N = 19 for Year 1 and 18 for 
Year 2), some items were identified as problematic through an empirical item analysis. The students 
in this tryout phases were small but were representative of the target population. Specifically, a few 
items were too difficult (proportion passing < 10%) or too easy (proportion passing > 95%) to be 
useful. Upon review of the content of the items by the panel of experts, adjustments were made to 
item wording, the distractors, and in some instances the entire item was replaced. 

2.4 Participants’ Background 

The influence of introducing the concept of scaling and providing hydrologic modeling case stud¬ 
ies and exercises was evaluated within two consecutive sessions of the Junior level Soil and Water 
Conservation Engineering course at Purdue University (ABE 325) through a static group pretest- 
posttest design. This course was selected because it is the main course for the environmental option 
in the Department of Agricultural and Biological Engineering (ABE) and is designed and taught by 
one of the authors. As a prerequisite for the course, students have to take an introductory course 
in soil science and a course in hydraulics (fluid mechanics). Participating students represented the 
range of students expected to benefit from using these tools and materials. Typical class size is 
18-25 students, with the majority being juniors and 88% of the participants being male. 

2.5 Year 1 Module Feedback 

After the first evaluation (Year 1: Fall semester of 2006), feedback was collected from students in 
informal focus groups to improve and refine the teaching materials (Table 1). Students and instructors 
recommended that more focus be given to the functional relationships (links) across hydrologic scales 
in an attempt to show the relevance of small scale (particle and laboratory) processes on hydrologic 
responses at larger scales (which is the scale at which decisions and policies are made). This link would 
trigger the students’ interest in hydrologic processes across the various scales, and provide the connec¬ 
tion between small- and large-scale processes and responses. The Module evaluation procedure in Year 1 
(Figure 2) was revised (Figure 3) and evaluated in the following course session (Year 2: Fall semester of 
2007) where considerable improvement in student learning and decision making skills was observed. 


3. MODULE COMPONENTS 


The educational module included different components including lecture materials, computer 
model user’s manuals, computer lab exercises, homework, and evaluation tests. All materials are 
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Comment 


Action or Change Implemented 


Although every scale was addressed (particle, lab, 
horizon, field, watershed), the relationships between 
scales were missing. Students were interested in 
learning how a process at the micro scale would affect 
or induce a response at the macro scale. 

Students didn’t see the relevance or importance of 
micro processes on macro scales where policies and 
decisions are made. This made them pay less attention 
to small scale processes. 

In Year 1, only Kamel® model was used but with 
computer lab after one lecture-hour. Students were 
concerned that the amount of lecture time was not 
enough to grasp the theory behind the model. The 
concepts of mobile and immobile water, micro and 
macro pores in the soil matrix, cracking network, as 
well as shrinkage and swelling were overwhelming or 
confusing. 

Students wondered how models like Kamel® could 
help improve their decision making skills. 

The link between the lecture and the computer lab 
was missing and students observed them as two 
independent activities. 

Some of the questions in the tests contained technical 
jargon that was not clear to some of the students. 


Additional lecture slides were developed to specifically 
address these issues. 


Instead of one lecture that was allocated for the module in 
the first year, three lectures were given in the second year. 
The first targets the general concept of scaling, the second 
targets the pedostructure concept in specific, and the third 
target the effect of scaling on decision making. 


More time was allocated for discussion of the implication of 
soil properties on policy and decision making. Also, WEPP® 
model was introduced to assist students in observing the 
effect of soil type, slope, field size, and the introduction of a 
buffer strip on runoff and erosion generation. 

The pre- and post- tests were changed and more attention 
was paid to the language used. 


Table 1: Summary of comments from feedback focus group following Year 7 Evaluation 
(N = 10). 


available online and can be downloaded from: http://cobweb.ecn.purdue.edu/~mohtar/ASABE- 
08workshop.htm . The following is a brief description of those components following the order in 
which they are used for the module (Figures 2 and 3): 

1 . Pretest: The pretest was designed to assess students’ initial level of knowledge about hydro- 
logic processes and their ability to apply their knowledge to simple decision making problems. 
Questions involved both conceptual and intuitive subjects, with different levels of difficulty. 
Additionally, student’s self assessment and LOC in their knowledge and decision-making skills 
were assessed. Both sets of these items were in the form of a Likert scale. Assessment of initial 
results (Year 1) showed that three questions in the pretest showed negative item discrimina¬ 
tion values and contributed to lowering the overall posttest scores compared to the pretest 
scores. That is, the high scoring group was responding to these questions incorrectly more 
often compared to the low scoring group. This may have been due to the complexity and level 
of difficulty of those questions, and in particular the distractor responses. This information was 
used in the evaluation feedback loop. Those questions were improved, eliminated or replaced 
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as deemed necessary following the item analysis. The modified test (Year 2) included seventeen 
questions and students took approximately 12 minutes to complete the assessment (Appendix 
1). Note that the assessment was meant to be brief and to be used in a variety of settings. 

2. Lecture Presentations: Urbanization, deforestation, and agricultural managment practices are 
all examples of environmental issues that impact hydrologic processes. These processes span 
a wide range of scales and orders of magnitude in space and time. From pore spaces within 
the soil (less than a mm) to large watersheds (tens of thousands of square km), each scale is 
governed by its unique set of physical and chemical laws and is characterized by its own ar¬ 
ray of assumptions and boundary conditions. Proper understanding of scaling and hydrologic 
processes is not usually explicitly incorporated in the teaching curriculum. The lecture presen¬ 
tations were designed to explain the concept of scaling as it relates to hydrologic modeling. 
This includes the interaction between spatial and temporal scales as well as the sensitivity of 
the scale selection on the dominant processes and nature of the problem at hand. This was 
followed by an introduction to the pedostructure concept, which will be used as a case study 
along with a computer simulation tool (Kamel®) with the objective of further clarifying the 
concept of scaling and its effect on hydrologic modeling. The effect of spatial and temporal 
variability also was explained followed by an introduction to WEPP®, a computer model that 
simulates sediment and runoff at the hillslope scale. 

3. Hands-on Computer Labs: Following lecture presentations, two computer labs were delivered. 
The first lab was assigned with the objective of increasing understanding of the effect of (1) 
soil variability; (2) evapotranspiration rate; and (3) rainfall intensity on hydrologic processes 
and responses. A presentation explaining the model (Kamel®) and showing qualitatively the 
short and long term effects of soil variability, evapotranspiration, and rainfall intensity was 
used to start the in-class lab session. Students then followed a step-by-step procedure to per¬ 
form several model simulations of a field with selected sets of soil types, evapotranspiration 
rates, and rainfall intensities and durations. The second computer lab used the WEPP® model 
to demonstrate the effect of slope, landuse, and soil type on runoff and sediment response. 
Students used a series of defined hillslopes on which they could test the effect of introducing 
sand, grass, and/or impermeable buffer strips on runoff and soil loss. For both labs, handouts 
were provided (in addition to being accessable online) to cover the step-by-step procedures 
and the homework problem statements. This procedure familiarized the student with the 
models and provided a foundation from which the students could complete the assigned 
homework. 

4. Posttest 1: Prior to leaving the in-class first computer lab session, students completed the first 
of two posttests (same questions as pretest). The objective of this posttest was to measure the 
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improvement in students’ understanding and decision making skills after having been through 
the lecture presentations and the in-class labs using the field level computer models. 

5. Homework: The homework exercises were designed to have the students apply their knowledge 
of process-based scaling (learned through the lectures and in-class lab exercises) to improve 
the understanding of the effects of soil texture and landuse variability, slope, evapotranspira- 
tion, and rainfall intensity at the field and watershed scales. The first homework (see Figure 3), 
which followed Lab 1, allowed students to simulate the differences in hydrologic response (us¬ 
ing Kamel®) between two evapotranspiration patterns, two rainfall events, and four soil texture 
types. The second homework, which followed Lab 2, allowed students to further explore the 
effect of changes in slope type, slope length, soil type, landuse, and placement of a buffer strip, 
using the WEPP® model. The introduction of watershed scale processes using the Kamel® and 
WEPP® models (both designed for field scale) was done by developing simplified watersheds as 
a spatial aggregation of multiple fields with variable spatial and rainfall properties. The spatial 
and temporal variability of the soil, landuse, slope, evapotranspiration, and rainfall intensity 
were assumed to represent the transformation from the field scale to the watershed scale. 
This assumption ignores additional processes that take place at the watershed scale including 
channel and stream processes. Thus, students were asked to perform various simulations to 
represent fields within the watershed, which opened discussions into how those results might 
combine at the watershed scale. 

6. Posttest 2: After completion of the homework, students were asked to complete the second 
posttest. This test was intended to measure changes in students’ understanding of scaling 
concepts and decision making skills after completing all aspects of the teaching module. 


4. RESULTS AND DISCUSSION 


The combined students’ performance in the enhanced learning and decision making questions 
were considered the main instrument to assess the effectiveness of the module. Self assessment 
and level of confidence also were considered in relationship to students’ performance. Evaluations 
of Year 1 and Year 2 described in the following two sections show the improvement in students’ 
knowledge and understanding. 

4.1 Fall 2006 Evaluation (Year 1) 

Year 1 evaluation results showed increased improvement in scores of pretest, posttest 1 and 
posttest 2. A summary of results obtained is provided in Table 2 and illustrates an increase in the 
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Yearl 


Test 

Mean' (%) 

Standard Deviation' (%) 

Self assessment Score 

Level of Confidence 

Pretest 

69.6 

13.3 

3.29 

3.77 

Posttest 1 

78.3 

12.7 

3.45 

3.95 

Posttest 2 

84.7 

13.9 

3.68 

4.06 



Yearl 



Test 

Mean 2 (%) 

Standard Deviation (%) 

Self assessment Score 

Level of Confidence 

Pretest 

53.3 

11.5 

3.45 

3.17 

Posttest 1 

60.5 

12.4 

3.82 

3.71 

Posttest 2 

62.4 

13.7 

4.12 

3.67 


'These scores are the outcome of analysis after removing results of three questions that gave negative item discrimination values. The real means 
including all questions for Year 1 are: 76.1 (pretest), 79.4 (posttest 1), and 77.7 (posttest 2). 

2 These scores are the outcome of the modified test in Year 2 with seventeen questions instead of thirteen in Year 1. Lower scores are the result of 
an increase in question difficulty and number rather than a change in student level from Year 1 to Year 2 

Table 2: Summary of Year 7 and Year 2 Evaluation results. 


mean score values from 69.6% in the pretest to 78.3% in the first posttest after students received 
the lecture and applied the computer model at the field scale. This score increased to 84.7% in the 
second posttest as students had more time and opportunites to apply the model at the watershed 
level. Similar trends were observed in the self assessment questions and level of confidence scores 
(Table 2). Dependent t-tests were used to judge the improvement and the associated effect sizes 
are reported. To assist in evaluating the magnitude of the differences in means, Cohen’s d (Cohen, 
1988) effect size (or standardized difference) also was computed, where 0.2, 0.5, and 0.8 are small, 
moderate, and large effects, respectively. The difference from pretest to posttest 1 was statistically 
significant (7(22) = 2.30, p < 0.05, d = 0.48). The difference from posttest 1 to posttest 2 was not 
significant (7(19) = 1.79, p > 0.05, d = 0.46). However, the moderate effect size in both cases sug¬ 
gests that there was a noteworthy improvement in performance. Note that data on three students 
were not available for the second analysis. Thus, lower statistical power for a smaller change could 
be one reason for the non-significant result from posttest 1 to posttest 2. 


4.2 Fall 2007 Evaluation (Year 2) 

Results of the 7-test showed statistically significant improvements (p < 0.05) and large effects 
(d > 0.80) on students’ self-assessment, learning, and level of confidence at addressing scaling 
related issues in Year 2. Table 3 summarizes the results of the d value of the 7-test based on the 
evaluation of 18 students who attended the entire module and completed all assessments. For the 
test scores, the differences from pretest to posttest 1 were statistically significant (7(18) = 2.37, 
p < 0.05, d = 0.60). The difference from posttest 1 and posttest 2 was not significant (7(18) = 0.51, 
p > 0.05, d = 0.15). 
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Self assessment 
Questions 

Knowledge (Learning & 
decision making combined) 

Level of Confidence 

Pre test to post test 1 

t (18) = 2.34 

t (18) = 2.37 

t (18) = 3.35 


p = 0.0320 

p = 0.0296 

p = 0.0038 


d = 0.62 

d = 0.60 

d = 0.77 

Post test 1 to post test 2 

t (18) = 1.59 

t (18) = 0.51 

t (18) = 0.06 


p = 0.1304 

p = 0.6183 

p = 0.9519 


d = 0.45 

d = 0.15 

d = 0.01 

Pre test to post test 2 

t (18) = 6.09 

t (18) = 3.20 

t (18) = 2.42 


p = <0.0001 

p = 0.0053 

p = 0.0270 


d = 1.40 

d = 0.73 

d = 0.6 


p < 0.05 was used to indicate statistical significance 


Table 3: Summary of Year 2 Evaluation results (Fall semester of 2007). 


For the Year 2 evaluation (Fall semester of 2007), average scores (correct answers) increased 
from 53.3% on the pretest to 60.5% and 62.4% in the posttest 1 and posttest 2, respectively (Figure 
4). This equates to approximately a 1.5 question improvement in correct answers. Average students’ 
level of confidence also increased from 3.17 in the pretest (on a scale of 5) to 3.71 and a slight drop 
(3.67) in posttest 1 and posttest 2, respectively (Figure 5). However, the difference between 3.71 
and 3.67 may merely reflect error and it is more likely that confidence remained stable. Moreover, 
students’ confidence in scaling-related issues increased after taking the module. This was identified 
by the increase in scores in the self-assessment section. On a scale of one to five, students raised 
their assessment of their own ability to understand scaling related issues from 3.45 in the pretest 
to 3.82 and 4.12 in posttests 1 and 2, respectively (Figure 6). 

Analysis of students’ scores from the Pretests of Year 1 and Year 2 showed that the effect of scale 
interactions and governing processes at smaller scales on hydrologic responses at larger scales were 
not fully understood by students. Reasons for this may be attributed to the limited exposure to 
scientific classes that address small scale processes and link their contribution and effect to larger 
scale responses. Following the module, students became more aware of the connection between 
scales and the effect of micro processes on large scales. 


5. SUMMARY AND CONCLUSION 


An educational module that incorporates the concept of process-based scaling into students’ 
curriculum was developed and has proven to positively impact students’ learning and their decision 
making capabilities. An outcome based evaluation procedure was applied to measure its effectiveness. 
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Figure 4: Average scores for the Year 2 evaluation (Fall semester of 2007). 
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Figure 5: Average level of confidence scores for the Year 2 evaluation (Maximum score = 
5, Fall semester of 2007). 


Results of a two-stage evaluation process showed a significant improvement in students’ learning 
and decision making skills (as defined in the methodology section). They also showed that the effect 
of interactions and governing processes at smaller scales to hydrologic responses at larger scales 
were not fully understood by students before taking the module. Students who complete the learning 
module may be more competent in and more prepared to deal with large-scale processes and scales. 
At the completion of the module, students are more aware of scaling effects and more appreciative 
of their contribution to large scale processes and effects. 

The module included a combination of lectures, homework assignments, and computer scenarios. 
Given the interdisciplinary nature of the topic and the wide range of departments and disciplines 
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Figure 6: Average self-assessment, or “self assessment” questions, scores for the Year 2 
evaluation (Maximum score = 5, Fall semester of 2007). 


involved in teaching multi-scale hydrologic processes, the module is designed with the flexibility 
to accommodate different environmental disciplines and levels of interventions. Successful imple¬ 
mentation of such programs should assist in enhancing student preparation for the future careers 
in environmental and natural resources engineering. The evidence provided in this project provides 
a foundation on which to build more evidence across program areas for the implementation of such 
curriculum programs. This project may serve as a model on which other such programs in various 
engineering areas can be constructed. Successful implementation of such programs can assist in 
enhancing student preparation for the future careers in environmental and natural resources en¬ 
gineering. Successful preparation of society’s future engineering professionals is paramount given 
the role engineering plays in society. 
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